Preserve indexer standalone-compactor flag from config

build_and_validate unconditionally overwrites indexer.enable_standalone_compactors from a separate top-level ConfigValue, whose default is false; this means setting indexer.enable_standalone_compactors: true in the indexer section is silently ignored unless QW_ENABLE_STANDALONE_COMPACTORS (or the top-level field) is also set. In practice, operators enabling standalone compaction through the documented indexer config will still run in legacy mode and may never start external compactor flows.

Useful? React with 👍 / 👎.

Validate planner endpoint, not just janitor presence

On compactor-only nodes, discovery succeeds as soon as any janitor connection exists, but this does not guarantee that the janitor exposes the compaction planner RPC (for example, janitor nodes running with standalone compactors disabled). In that case startup proceeds with a client to a node that cannot serve report_status, causing repeated runtime failures instead of a clear startup error.

Useful? React with 👍 / 👎.

Enable split cache when indexer runs without compactor

This condition now falls back to IndexingSplitCache::no_caching() unless both indexer and compactor services are enabled, which disables the indexer split cache in common indexer-only deployments. Before this change, IndexingService::new always opened the cache from indexer_config; with the new wiring, legacy merge pipelines on indexer-only nodes repeatedly re-download splits instead of reusing local cache, causing a significant performance and bandwidth regression.

Useful? React with 👍 / 👎.

Mark one-shot compaction pipelines without actor termination

This completion test only succeeds when every child actor reports Health::Success, but the downloader/executor/packager/uploader/publisher actors spawned for a compaction task are the normal long-lived actors and do not exit after processing a single merge message. After a successful publish they therefore stay Healthy until the progress heartbeat check eventually marks idle actors unhealthy, so the task is never reported as Completed (and may be reported failed), leaking compactor slots and preventing the planner from receiving successful acknowledgements.

Useful? React with 👍 / 👎.

Remove adopted heartbeat splits from pending vectors

When the planner has already scanned a split into needs_compaction and later receives an in-progress heartbeat for an unknown task (the recovery case described here), this only removes the split ID from needs_compaction_split_ids; the SplitMetadata remains in the needs_compaction vector. plan_partition operates on that vector, so the next planning tick can enqueue a second merge for the same split while the worker is still reporting it in progress. Remove the matching metadata from needs_compaction as well, or make planning skip IDs that are already in flight.

Useful? React with 👍 / 👎.

Apply exclusions before immature split early returns

For compaction planner scans against the file-backed metastore, queries combine retain_immature with excluded_split_ids, but split_query_predicate returns from the maturity check above before reaching this new exclusion block. Once the first SCAN_PAGE_SIZE immature splits are already tracked, list_splits keeps returning that same excluded page, ingest_splits skips it, and the planner never advances to later splits. Move this exclusion ahead of the maturity return path or avoid returning early there.

Useful? React with 👍 / 👎.

Keep Parquet merge pipelines until compactors support them

When enable_standalone_compactors disables the local merge scheduler, this also returns None for metrics/Parquet indexes, so their publishers no longer feed a ParquetMergePlanner. The new standalone compactor only plans MergeOperations from the regular splits table and does not process metrics_splits/ParquetMergeTasks, so metrics indexes in standalone mode will accumulate unmerged Parquet splits indefinitely. Keep the Parquet merge pipeline active or add equivalent standalone Parquet compaction before dropping this mailbox.

Useful? React with 👍 / 👎.

Run finalize merge policy in standalone compaction

With standalone compactors enabled, indexers no longer spawn local merge pipelines, so source shutdown no longer sends RunFinalizeMergePolicyAndQuit; this planner path only calls operations(). For finite or cold sources that rely on max_finalize_merge_operations, the final merge policy is never applied and straggler splits below the normal merge factor remain unmerged. Add a finalize path in the standalone planner or keep local finalization for those sources.

Useful? React with 👍 / 👎.

Include compactor supervisor in liveness checks

This stores the compactor supervisor separately, but the health routes still only receive the indexer and janitor mailboxes, and CompactorSupervisor has no Healthz wiring. On compactor-only nodes, if the supervisor actor fails, /health/livez continues to return healthy because no enabled service is checked, leaving orchestration unaware that compaction has stopped. Wire the supervisor into the liveness path like the indexer/janitor services.

Useful? React with 👍 / 👎.

Elect a single compaction planner per cluster

In clusters with more than one janitor, this starts an independent in-memory CompactionPlanner on every janitor, while compactor-only nodes use a balanced Janitor channel for report_status. A task can be assigned by one planner and reported to another, and each planner scans the same Published immature splits without shared state, so duplicate compactions and spurious timeouts/reassignments are likely in the default multi-janitor topology. Use a single elected planner or persist/shard assignment state before enabling multiple planners.

Useful? React with 👍 / 👎.

Do not alias current doc mapper to old split UIDs

When an index has old immature splits from a previous doc mapping, get_for_split passes that old doc_mapping_uid here, but this code fetches only the current index metadata and stores the current doc mapper under the old UID. The resulting assignment serializes self.config.doc_mapping, so standalone compactors can merge old-schema splits with the new schema after a mapping update. Verify that the fetched config's UID matches the split UID, or retain/fetch the historical doc mapping for that UID.

Useful? React with 👍 / 👎.

Drop tracked splits once they become mature

For sparse sources or partitions that never reach the merge factor, a split can age past its maturation period while sitting in needs_compaction. The merge policies skip mature splits by leaving them in splits, so operations() returns empty here and the later is_empty() cleanup never removes their IDs from needs_compaction_split_ids; those stale IDs are then excluded on every metastore scan and can grow unbounded on long-running planners. Re-evaluate and remove now-mature splits from the tracked set before breaking out.

Useful? React with 👍 / 👎.

Validate assignment index UID before using it

When a compactor receives a MergeTaskAssignment without index_uid (the proto field is optional on the wire, and build_compaction_pipeline already treats this as a recoverable error), this unwrap runs before that validation and panics the supervisor while building metric labels. A malformed planner response should be logged as a failed spawn or rejected through the existing ok_or_else path rather than taking the compactor actor down.

Useful? React with 👍 / 👎.

Refresh cached index settings after metadata updates

If an operator updates an index's indexing_settings or retention_policy without changing the doc mapping UID, this cache hit keeps returning the old IndexEntry forever. The standalone planner then continues planning with the stale merge policy and sends compactors the stale retention policy until the janitor restarts, even though update_index supports changing these fields at runtime; cache on an index metadata version or periodically refetch instead of keying freshness only on doc_mapping_uid.

Useful? React with 👍 / 👎.

Reject zero compactor upload concurrency

When compactor.max_concurrent_split_uploads is set to 0, deserialization accepts it and the compactor later constructs merge uploaders with a zero-permit semaphore, so the first packaged split waits forever for an upload permit and the task remains in progress with its slot stuck. Make this NonZeroUsize or add validation so a bad compactor config fails at startup instead of deadlocking compaction.

Useful? React with 👍 / 👎.

that feels like something which can now go to zero if max_concurrent_split_uploads=1 on the "there is a merge_scheduler" path, and cause issues down the line

Great catch, you're right, this shouldn't have changed

(i'm not sure if that shouldn't have changed, the "there is no merge_scheduler path" might want its zero)

i'm not entirely convinced by the use of a partial index here. Moving from Staged->Published and Published->MarkedForDeletion both need to write to this index, and Published should be the majority of splits at any given point in time, i don't think having a partial index here gains us much

iirc this used to not be the property we settled upon, but i think this is a better level of atomicity and rollback capabilities. If a migrations worked there is no reason to rollback because the next one failed

❤️

there are other places we should do that i think, i recall seeing queries with dozens (hundreds?) of parameter, for split deletion by id or something like that

(not something to do in this PR, but we should track that somewhere)